Skip to content

fix: strip cache_breakpoint in LiteLLM path and expand Anthropic prefixes#5914

Open
superShen0916 wants to merge 2 commits into
crewAIInc:mainfrom
superShen0916:fix/llm-routing-improvements
Open

fix: strip cache_breakpoint in LiteLLM path and expand Anthropic prefixes#5914
superShen0916 wants to merge 2 commits into
crewAIInc:mainfrom
superShen0916:fix/llm-routing-improvements

Conversation

@superShen0916

@superShen0916 superShen0916 commented May 23, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown

Summary

Two small fixes to improve LLM provider routing robustness:

1. Strip cache_breakpoint in LiteLLM fallback path (Fixes #5886)

_format_messages_for_provider was not stripping cache_breakpoint flags before sending messages to providers through the LiteLLM path. This caused errors like:

GroqException - 'messages.0': property 'cache_breakpoint' is unsupported

BaseLLM._format_messages already handles this for native providers (OpenAI, Anthropic, etc.), but the LiteLLM fallback path uses its own method which missed this step. The fix strips the flag from message copies before they reach LiteLLM.

2. Expand Anthropic model prefix matching (Fixes #5893)

Added anthropic-- to the Anthropic model prefix list in _matches_provider_pattern. This allows custom deployments with names like anthropic--claude-sonnet-4 to be recognized as Anthropic models and routed to the native provider.

Changes

  • lib/crewai/src/crewai/llm.py: Strip cache_breakpoint from message copies in _format_messages_for_provider (the LiteLLM fallback path) before provider-specific formatting
  • lib/crewai/src/crewai/llm.py: Add anthropic-- to Anthropic model prefix detection

Test plan

  • Existing tests pass: TestCacheMarkerHelpers, TestBaseFormatDoesNotMutate, TestNonAnthropicStripsMarker
  • The message copy approach preserves caller message lists (no mutation) — consistent with the pattern used in BaseLLM._format_messages

Risk

Low. Changes are limited to stripping an internal-only flag from API-bound messages (already done in the native path) and expanding a prefix allowlist by one entry.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Improved compatibility with Anthropic/Claude-style model naming so more provider model names are recognized.
    • Sanitized outgoing message payloads to strip cache-break markers before sending, preventing marker leakage.
    • Ensured per-message copies are used during formatting to avoid unintended mutation.

Review Change Stack

@claude
fix: strip cache_breakpoint in LiteLLM path and expand Anthropic pref…
303d166 
…ixes

- Strip cache_breakpoint flags in _format_messages_for_provider to fix
  errors with Groq and other LiteLLM-routed providers that reject
  unknown message fields (e.g., 'cache_breakpoint is unsupported').
  BaseLLM._format_messages already handles this for native providers.
- Add 'anthropic--' to the Anthropic model prefix list so custom
  deployments with names like 'anthropic--claude-...' are recognized.

Fixes crewAIInc#5886
Fixes crewAIInc#5893

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented May 23, 2026 •
edited
Loading

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6114041e-beb1-4886-9675-59f7f3f64238

📥 Commits

Reviewing files that changed from the base of the PR and between 303d166 and bf1d95b.

📒 Files selected for processing (1)
  • lib/crewai/src/crewai/llm.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • lib/crewai/src/crewai/llm.py
📝 Walkthrough

Walkthrough

Adds support for an additional Anthropic model-name prefix (anthropic--) when matching Anthropic/Claude providers, and sanitizes LiteLLM outgoing messages by shallow-copying each message and stripping cache_breakpoint markers before provider-specific formatting.

Changes

LiteLLM Provider Compatibility

Layer / File(s) Summary
Expanded Anthropic model prefix validation
lib/crewai/src/crewai/llm.py		 Anthropic/Claude provider matching now accepts anthropic-- in addition to claude- and anthropic..
Cache breakpoint sanitization for LiteLLM
lib/crewai/src/crewai/llm.py		 _format_messages_for_provider shallow-copies each message dict and applies strip_cache_breakpoint to remove cache_breakpoint flags before provider-specific formatting.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

  • crewAIInc/crewAI#5774: Earlier changes to LiteLLM message sanitization and Anthropic/Claude routing that this PR builds upon.

Suggested labels

size/L

Poem

🐰 I nibbled prefixes in the night,
I pruned the flags that gave a fright,
Anthropic names now pass the gate,
Messages tidy, clean, and straight,
Hooray — LiteLLM hops light!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes both main changes: stripping cache_breakpoint in the LiteLLM path and expanding Anthropic prefixes.
Linked Issues check ✅ Passed The changes address both linked issues: stripping cache_breakpoint for non-Anthropic providers [#5886] and expanding Anthropic model prefix matching to include 'anthropic--' [#5893].
Out of Scope Changes check ✅ Passed All changes in lib/crewai/src/crewai/llm.py are directly scoped to the two issues: cache_breakpoint stripping and Anthropic prefix expansion.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@leavedrop leavedrop mentioned this pull request May 25, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] approved these changes

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

1 participant

@superShen0916